Stackelberg Actor-Critic: Game-Theoretic Reinforcement Learning Algorithms

The hierarchical interaction between the actor and critic in actor-critic based reinforcement learning algorithms naturally lends itself to a game-theoretic interpretation. We adopt this viewpoint model as two-player general-sum game with leader-follower structure known Stackelberg game. Given abstraction, we propose meta-framework for where leader player follows total derivative of its objective instead usual individual gradient. From theoretical standpoint, develop policy gradient theorem refined update provide local convergence guarantee equilibrium. an empirical demonstrate via simple examples that dynamics study mitigate cycling accelerate compared given cost structures induced by formulations. Finally, extensive experiments on OpenAI gym environments show always perform at least well often significantly outperform standard algorithm counterparts.